Unix & Linux

Unix & Linux

contact: ludwig@univ-brest.fr

📖 Linux is User-Friendly too, it’s just more peaky who its friends are...

1. Why Linux ?

  • Write programs that does one thing and does it well.
  • Write programs that work together.
  • Write programs that handle text streams, because that is a universal interface.

2. Why learn Linux basic command lines ?

  • Simple : do one thing
  • Flexible : built for reuse
  • Versatile : can do anything/everything
  • Fast : no graphics overload
  • Ubiquitous : available on all machines— Permanent : First released in 1971

3. shell (command-line interpreter)

A Unix terminal is a command-line interpreter or shell that provides a traditional Unix-like command line user interface. Users direct the operation of the computer by entering commands as text for a command line interpreter to execute, or by creating text scripts of one or more such commands.

4. bash (one type of shell, like sh, ksh, zsh,...)

bash is a command processor that typically runs in a text window, where the user types commands that cause actions.

5. Linux Shortcuts

  • Previous command : Key
  • Next command : Key
  • Autocompletion : (TAB) Key
Copy+Paste:
  • Unix/Unix:
    1. Select text
    2. middle-click (wheel button) where you want to paste
  • Windows/Unix:
    1. Select text and Ctrl-C in Windows
    2. middle-click (wheel button) where you want to paste in Unix
  • Unix/Windows:
    1. Select text in Unix
    2. Ctrl-V to paste in Windows

6. echo (print a message)

Command 6.1

echo

bash

Command 6.2

echo "ATGC"

bash

Command 6.3

echo -n "ATGC"

bash

Command 6.4

echo -e  ">seq\nATGC"

bash

Command 6.5

echo -e  ">seq\tATGC"

bash

Command 6.6

echo -e -n  ">seq\nATGC"

bash

7. filesystem

  • 'home' (See ~)
  • 'root' /
/
+-- /home/
|   +-- /home/tludwig/
|       +-- /home/tludwig/seqlink/
|       |   +-- /home/tludwig/seqlink/SEQLinkage-master/
|       |   +-- /home/tludwig/seqlink/SEQLinkage-master.zip
|       +-- /home/tludwig/linux/
|       |   +-- /home/tludwig/linux/installed.pkg
|       +-- /home/tludwig/file1.txt
|       +-- /home/tludwig/file2.txt
+-- /etc
+-- /usr

bash

8. path

  • A Unix/Linux path name is a text string made up of one or more names separated by slashes
  • Destination object may be file, directory, or other
  • . current directory
  • .. parent directory
  • / root directory
  • ~ the home directory of the current user
Relative paths vs Absolute paths
  • Relative : the path relative to the current directory (will word from the current directory)
  • Absolute : the path from the root directory (will work from anywhere)

Pasted image 20230921093225.png

  • /home/user/directory/file.txt : absolute path
  • directory/file.txt: path relative to the current directory. Same as ./directory/file.txt

9. pwd (Print working directory)

Command 9.1 Where am I ?

pwd

bash

10. ls (Lists files and folders)

Command 10.1

ls

bash

Command 10.2

ls .

bash

Command 10.3

ls /

bash

Command 10.4

ls ../

bash

Command 10.5

ls ../..

bash

Command 10.6

ls -l /

bash

Command 10.7 ⚠

ls does_not_exists

bash

file globbing

* means anything

Command 10.8

ls /home/* #all files and directories in /home/

bash

Commande 10.9

ls /etc/*.conf #all files in /etc/ that end with .conf

bash

Command 10.10

ls -d /home/*i* #all /home subdirectories that contain i

bash

11. cd (Change Directory)

Command 11.1

cd ..

bash

Command 11.2

cd .

bash

Command 11.3

cd 

bash

Command 11.4

cd ~

bash

Command 11.5

cd /

bash

Command 11.6

cd -

bash

12. mkdir (make directory)

Command 12.1

mkdir dir1 dir2 dir3
ls

bash

Command 12.2 ⚠

mkdir a/b/c/d

bash

Command 12.3

mkdir -p a/b/c/d
ls
ls a
ls a/b/c

bash

13. cp (copy files or folders)

Command 13.1

cp /home/tludwig/master2ggb/file1.txt file1.txt
ls

bash

Command 13.2

cp file1.txt dir1
ls dir1

bash

Command 13.3

cp file1.txt dir1/file2.txt
ls dir1

bash

Command 13.4

cp dir1 dir2

bash

Command 13.5

cp -r dir1 dir2
ls dir2

bash

14. rmdir (remove directory)

rmdir dir1 dir2 dir3 a
ls

bash

15. rm (remove files or folders)

Command 15.1

rm file1.txt

bash

Command 15.2 ⚠

rm does_not_exits.txt

bash

Command 15.3

rm -f does_not_exits.txt

bash

Command 15.4 ⚠

rmdir a

bash

Command 15.5

rm -rf a
ls

bash

Command 15.6

rm dir1

bash

Command 15.7

rm -rf dir1 dir2

bash

16. seq (Create a sequence)

Command 16.1

seq 10

bash

Command 16.2

seq 2 9

bash

Command 16.3

seq 2 3 9

bash

17. cat (conCATenate files)

Preparation

cp /home/tludwig/master2ggb/file1.txt .
cp /home/tludwig/master2ggb/file2.txt .

bash

Command 17.1

cat file1.txt

bash

Command 17.2

cat file2.txt

bash

Command 17.3

cat file1.txt file2.txt

bash

Command 17.4

cat -n file1.txt file2.txt

bash
Most important thing you have to understand today

18. standard input

Where the data are read from (default : keyboard ⌨)

  • < uses a file as standard input.

19. standard output (stdout)

Where the results are written (default : screen )

  • > redirects standard output to a file (and overwrites the file if it already exists).
  • >> Appends standard output to a file.

Sans redirection

ls /root /bin

bash

Command 19.1

ls /root /bin > stdout.txt
cat stdout.txt

Command 19.2

echo "new data" > stdout.txt
cat stdout.txt

Command 19.3

ls /root ~ >> stdout.txt
cat stdout.txt

20. standard error (stderr)

Where the error messages are written (default : screen)

  • 2> redirects standard error to a file (and overwrites the file if it already exists)

Command 20.1

ls /root /bin 2> stderr.txt
cat stderr.txt

Command 20.2

ls /root /bin  > stdout.txt 2> stderr.txt
cat stdout.txt
cat stderr.txt

21. redirection (stdout | stdin )

  • | (pipe : AltGR+6 for normal people Alt+Maj+L for Mac users) redirects standard output of a program as standard input of another.

Sans redirection

ls /

Command 21.1

ls / | cat -n

Command 21.2

echo "1601627*5" | bc

To sum up

commandA < mydata.txt   # uses the content of the file named mydata.txt as the input of commandA (you probably won't use that)
commandA > output.txt   # writes the results of commandA in the file named output.txt
commandA 2> message.log # writes the messages of commandA (mostly errors, but also all non results texts) in the file named message.log 
commandA | commandB     # passes the results of commandA as the input of commandB (called "piping")

22. wc (Word Count)

Command 22.1

cat /etc/libaudit.conf
wc /etc/libaudit.conf

bash

Command 22.2

seq 1000000 | wc -c

bash

Command 22.3

seq 1000000 | wc -l

bash

Command 22.4

echo | wc -c

bash

Command 22.5

echo -n | wc -c

bash

23. stop input stdin : Ctrl-D

bc
5*5
2+9

bash

⌨ <ctrl+D>

24. stop a processus Ctrl-C

seq 10000000

bash

⌨<ctrl+C>

25. head (output the first part of files)

Command 25.1

seq 10000 | head

bash

Command 25.2

seq 10000 | head -3

bash

26. tail (output the last part of files)

Command 26.1

seq 10000 | tail

bash

Command 26.2

seq 10000 | tail -3

bash

Command 26.3

cat /home/tludwig/master2ggb/table.tsv

bash

Command 26.4

cat /home/tludwig/master2ggb/table.tsv | tail -n+2

bash

27. paste (merge lines of files)

Preparation

seq 1 10 > a 
cat a 
seq 2 12 > b 
cat b 

bash

Command 27.1

paste a b

bash

Command 27.2

seq 20 | paste - -

bash

Command 27.3

seq 20 | paste - - -

bash

28. cut (extract sections from each line of files)

Without cut

seq 1 100 | paste - - - - -

bash

Command 28.1

seq 1 100 | paste - - - - - | cut -f 3

bash

Command 28.2

seq 1 100 | paste - - - - - | cut -f 3-

bash

Command 28.3

seq 1 100 | paste - - - - - | cut -f 2-4

bash

Command 28.4

seq 1 100 | paste - - - - - | cut -f -2

bash

Command 28.4

seq 1 100 | paste - - - - - | cut -f 1,5

bash

Command 28.5

echo "ABCDE" | cut -c 2

bash

Command 28.6

echo "ABCDE" | cut -c 2,5

bash

Command 28.7

echo "ABCDE" | cut -c 2-5

bash

Command 28.8

echo "AA:BB:CC:DD:EE:FF" | cut -d':' -f2

bash

29. sort (sort lines of text files)

Command 29.1

seq 1 25 | sort

bash

Command 29.2

seq 1 25 | sort -n

bash

Command 29.3

seq 1 25 | sort -n -r

bash

Command 29.4

cat /home/tludwig/master2ggb/chrom.list

bash

Command 29.5

cat /home/tludwig/master2ggb/chrom.list | sort

bash

Command 29.6

cat /home/tludwig/master2ggb/chrom.list | sort -n

bash

Command 29.7

cat /home/tludwig/master2ggb/chrom.list | sort -V

bash

Command 29.8

cat /home/tludwig/master2ggb/table.tsv

bash

Command 29.9

cat /home/tludwig/master2ggb/table.tsv | tail -n+2 | sort -t $'\t' -k2n

bash

Command 29.10

cat /home/tludwig/master2ggb/table.tsv | tail -n+2 | sort -t $'\t'  -k3r -k2n

bash

30. uniq (report or omit repeated lines)

Without uniq

echo -e "A\nA\nA\nB\nB\nA"

bash

Command 30.1

echo -e "A\nA\nA\nB\nB\nA" | uniq

bash

Command 30.2

echo -e "A\nA\nA\nB\nB\nA" | sort | uniq

bash

Command 30.3

echo -e "A\nA\nA\nB\nB\nA" | sort -u

bash

Command 30.4

echo -e "A\nA\nA\nB\nB\nA" | sort | uniq -c

bash

Command 30.5

echo -e "A\nA\nA\nB\nB\nA" | sort | uniq -c | sort -n

bash

31. grep (print lines matching a pattern

Without grep

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT"

bash

Command 31.1

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep AT

bash

Command 31.2

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep GAATTC

bash

Command 31.3

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep -v ">"

bash

Command 31.4

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep -B1 GAATTC

bash

Command 31.5

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep -A1 gene2

bash

Command 31.6

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep -E '(GAATTC|CCT)' 

bash

Command 31.7

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep -o A

bash

Command 31.8

echo -e ">gene1\nATGGAATTCAAAA\n>gene2\nCCCTGCTGATCGATCGATCGT" | grep -o A | wc -l

bash

32. more / less (file perusal)

seq 1000000 | more
seq 1000000 | less

bash
  • <enter> to skip 1 line
  • <space> to skip 1 page
  • q to quit
  • arrows in less but not in more

33. touch (Change file timestamp, or create a file)

touch new_file
ls -l

bash

34. find (search for files in a directory hierarchy)

Command 34.1

find ~ -type f

bash

Command 34.2

find /etc -type f -name "*.conf"

bash

35. tr (Translate, squeeze, and/or delete characters)

Command 35.1

echo "ATAAAAACGACTTTTGA" | tr "T" "U"

bash

Command 35.2

echo "ATAAAAACGACTTTTGA" | tr -s "A"

bash

Command 35.3

echo "ATAAAAACGACTTTTGA" | tr -d "A"

bash

Command 35.4

echo "ATAAAAACGACTTTTGA" | tr "[ATGC]" "[TACG]"

bash

36. rev (reverse lines characterwise)

echo "ATAAAAACGACTTTTGA" | tr "[ATGC]" "[UACG]" | rev

bash

37. gzip (compress files)

Without gzip

seq 100000 | wc -c

bash

Command 37.1

seq 100000 | gzip | wc -c

bash

Without gzip

cat file1.txt

bash

Command 37.2

gzip file1.txt
cat file1.txt.gz

bash

38. gunzip / zcat (expand files)

Command 38.1

zcat file1.txt.gz

bash

Command 38.2

gunzip file1.txt.gz

bash

Command 38.3

seq 10 | gzip | gunzip 

bash

39. diff / sdiff (compare files line by line)

Preparation

seq 20 > a 
seq 10 30 > b 

bash

Command 39.1

diff a b

bash

Command 39.2

sdiff a b

bash